A Cascaded Machine-Learning Pre-Filter and Adaptive LLM Re-Ranking Architecture for Financial News Importance Scoring

Abstract

Financial news is published faster than any person can read it. This paper presents IntrAIdify, a live financial news ranking system built as a two-stage cascade: a machine learning ensemble (Linear Regression, Random Forest, XGBoost, LightGBM) first filters incoming headlines down to the top 10, and a large language model (LLM) then re-ranks those 10 using a softmax mechanism that adaptively weights the ML and LLM scores per headline. Three smaller signals - a keyword lexicon, a topic-relevance score, and an actionability score - add further weight to the final ranking. We evaluate the system on a full 398-day held-out test set in two ways: how well it ranks the full list of headlines, and how well the LLM re-ranks the top 10 the ML ensemble already chose. On the full-list test, the plain ML ensemble significantly outperforms the full pipeline. On the top-10 test, the LLM re-ranking shows no significant improvement over the ML ensemble\'s own ordering. We report both results plainly, discuss what they mean for the architecture, and argue that measuring a reranker within its actual candidate set, not just across the full list, is the correct way to evaluate any cascaded ranking system.

Introduction

This paper presents IntrAIdify, a live financial news ranking system designed to help traders, analysts, and investors prioritize important financial news from the overwhelming volume of daily headlines. Instead of predicting market sentiment or stock price movement, the system focuses on ranking news headlines by importance using a retrieve-then-rerank architecture, a common approach in information retrieval. A four-model machine learning ensemble first filters incoming headlines to a high-recall top-10 candidate set, after which a Large Language Model (LLM) reranks these candidates using an adaptive temperature-scaled softmax mechanism. Three additional signals—keyword salience, topic relevance, and lexical-semantic actionability—further refine the final ranking.

A major contribution of the work is its evaluation methodology. The authors argue that reranking systems should not be judged solely by their performance over the entire headline pool, since the LLM only processes the candidates selected by the first-stage model. Therefore, the system is evaluated using both conventional ranking metrics and a cascade-specific metric that isolates the LLM's contribution within the selected top-10 headlines. Experiments conducted on 398 held-out test days showed that neither the LLM reranker nor the auxiliary scoring signals significantly improved performance over the baseline machine learning ensemble. The paper openly reports these null findings, supported by paired statistical significance tests and component-wise ablation studies, demonstrating transparency regarding the effectiveness of each system component.

Unlike most previous financial NLP research, which focuses on sentiment analysis or stock price prediction, IntrAIdify formulates the problem as a learning-to-rank task. Prior studies have shown that machine learning and deep learning models such as LSTM, Random Forest, XGBoost, and sentiment analysis techniques can improve market prediction, while information retrieval research has demonstrated the effectiveness of retrieve-then-rerank architectures using transformer-based rerankers and LLMs. Building on these concepts, IntrAIdify applies the cascade framework specifically to financial headline importance ranking.

The system is implemented as a continuously running FastAPI service comprising three concurrent processes: an RSS news scraper that updates rankings every five minutes, a file watcher that detects changes in the top-ranked headlines and sends email alerts, and API handlers that provide rankings and manage user subscriptions. The ranking pipeline processes each batch of incoming headlines through seven sequential stages: text construction, TF-IDF vectorization, machine learning ensemble scoring, top-10 candidate selection, computation of auxiliary scores, adaptive LLM reranking with score fusion, and final normalization to generate the published ranking.

For training and evaluation, the system uses two labeled datasets: a Reddit world news dataset ranked by community votes and paired with Dow Jones market movement labels, and a financial news sentiment dataset based on State Bank of India news articles. During live operation, IntrAIdify continuously collects financial news from multiple RSS sources including Reuters, Bloomberg, CNBC, Yahoo Finance, Investing.com, Financial Times, Business Insider, Dow Jones, and the New York Times Business section.

Each headline is represented using four independent TF-IDF vectorizers. The first-stage ensemble combines predictions from four machine learning models: Linear Regression, Random Forest Regressor, XGBoost Classifier, and LightGBM Regressor. Their normalized outputs are equally weighted to produce an overall ensemble score that determines the top-10 candidate headlines.

Conclusion

This paper introduced IntrAIdify, a live financial news ranking system built as a two-stage cascade: an ML ensemble filters headlines down to the top 10, and an LLM re-ranks them. We evaluated the system on the full 398-day held-out test set in two ways: how well it ranks the full list of headlines, and how well the LLM re-ranks the top 10 the ML ensemble already picked. Both checks point the same way: the plain ML ensemble significantly outperforms the full pipeline on the full list (p = 0.0015 Spearman, p = 0.029 NDCG@10), and the LLM\'s re-ranking of the top 10 shows no significant improvement over the ML ensemble\'s own ordering. In plain terms: as currently built, the LLM re-ranking step and the auxiliary keyword/topic/actionability scores do not improve ranking quality over the ML ensemble alone. We report this honestly rather than reframing it, and we still see value in measuring both the full-list and candidate-set-restricted view for any cascaded ranking system, since in general the two can disagree even though they agreed here. Future work includes: (i) correcting the identified XGBoost class-probability selection and the LLM neutral-placeholder softmax interaction, which may be diluting whatever real signal the LLM stage has; (ii) extending the offline evaluation to include article summary text, matching the production system\'s actual input; (iii) conducting a live, prospective evaluation of the recency-decay component, which cannot be meaningfully assessed using historical data alone; (iv) generalizing the evaluation to other markets, asset classes, and time periods; and (v) revisiting the softmax blending weights and the auxiliary signal weights, since the current fixed weights (0.15, 0.15, 0.15, 0.20) for the auxiliary terms were not themselves tuned or validated against ranking quality.

References

[1] M. Verma, R. Jain, and S. Monga, \"Comparative analysis of machine and deep learning models with text embeddings for sentiment analysis,\" Review of Computer Engineering Research, vol. 13, no. 1, pp. 1-35, 2026. [2] S. R. Khonde, S. S. Virnodkar, S. B. Nemade, M. A. Dudhedia, B. Kanawade, and S. H. Gawande, \"Sentiment analysis and stock data prediction using financial news headlines approach,\" Revue d\'Intelligence Artificielle, vol. 38, no. 3, pp. 999-1008, 2024. DOI: 10.18280/ria.380325. [3] J. Deveikyte, H. Geman, C. Piccari, and A. Provetti, \"A sentiment analysis approach to the prediction of market volatility,\" Frontiers in Artificial Intelligence, vol. 5, art. 836809, 2022. DOI: 10.3389/frai.2022.836809. [4] M. Davidovic and J. McCleary, \"News sentiment and stock market dynamics: A machine learning investigation,\" Journal of Risk and Financial Management, vol. 18, no. 8, art. 412, 2025. DOI: 10.3390/jrfm18080412. [5] Aaron7sun. \"Daily news for stock market prediction\" [Dataset]. Kaggle. [Online]. Available: https://www.kaggle.com/datasets/aaron7sun/stocknews (accessed via mirror at https://www.kaggle.com/datasets/tanishqdublish/stock-market-predictions). [6] P. Kumar. \"Sentiment intensity for news articles of SBI\" [Dataset]. Kaggle. [Online]. Available: https://www.kaggle.com/datasets/parshantkumar2033/sentiment-intensity-for-news-articles-of-sbi [7] K. Sparck Jones, \"A statistical interpretation of term specificity and its application in retrieval,\" Journal of Documentation, vol. 28, no. 1, pp. 11-21, 1972. [8] L. Breiman, \"Random forests,\" Machine Learning, vol. 45, no. 1, pp. 5-32, 2001. [9] T. Chen and C. Guestrin, \"XGBoost: A scalable tree boosting system,\" in Proc. 22nd ACM SIGKDD Int. Conf. Knowledge Discovery and Data Mining, San Francisco, CA, USA, 2016, pp. 785-794. [10] G. Ke, Q. Meng, T. Finley, T. Wang, W. Chen, W. Ma, Q. Ye, and T.-Y. Liu, \"LightGBM: A highly efficient gradient boosting decision tree,\" in Advances in Neural Information Processing Systems 30 (NeurIPS 2017), Long Beach, CA, USA, 2017, pp. 3146-3154. [11] K. Jarvelin and J. Kekalainen, \"Cumulated gain-based evaluation of IR techniques,\" ACM Transactions on Information Systems, vol. 20, no. 4, pp. 422-446, 2002. [12] F. Wilcoxon, \"Individual comparisons by ranking methods,\" Biometrics Bulletin, vol. 1, no. 6, pp. 80-83, 1945. [13] T. Cover and P. Hart, \"Nearest neighbor pattern classification,\" IEEE Transactions on Information Theory, vol. 13, no. 1, pp. 21-27, 1967. [14] A. Jain, K. Nandakumar, and A. Ross, \"Score normalization in multimodal biometric systems,\" Pattern Recognition, vol. 38, no. 12, pp. 2270-2285, 2005. [15] J. S. Bridle, \"Probabilistic interpretation of feedforward classification network outputs, with relationships to statistical pattern recognition,\" in Neurocomputing, Berlin, Germany: Springer, 1990, pp. 227-236. [16] Gemini Team, Google, \"Gemini: A family of highly capable multimodal models,\" arXiv preprint arXiv:2312.11805, 2023. [17] G. Salton, A. Wong, and C. S. Yang, \"A vector space model for automatic indexing,\" Communications of the ACM, vol. 18, no. 11, pp. 613-620, 1975. [18] Y. Koren, \"Collaborative filtering with temporal dynamics,\" Communications of the ACM, vol. 53, no. 4, pp. 89-97, 2010. [19] R. Nogueira and K. Cho, \"Passage re-ranking with BERT,\" arXiv preprint arXiv:1901.04085, 2019. [20] Z. Qin, R. Jagerman, K. Hui, H. Zhuang, J. Wu, L. Yan, J. Shen, T. Liu, J. Liu, and D. Metzler, \"Large language models are effective text rankers with pairwise ranking prompting,\" in Findings of the Association for Computational Linguistics: NAACL 2024, 2024, pp. 1504-1518. [21] T.-Y. Liu, \"Learning to rank for information retrieval,\" Foundations and Trends in Information Retrieval, vol. 3, no. 3, pp. 225-331, 2009. [22] T. G. Dietterich, \"Ensemble methods in machine learning,\" in Proc. Int. Workshop on Multiple Classifier Systems, Berlin, Germany: Springer, 2000, pp. 1-15. [23] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay, \"Scikit-learn: Machine learning in Python,\" Journal of Machine Learning Research, vol. 12, pp. 2825-2830, 2011.

Copyright

Copyright © 2026 Viswanath Parashuram Yadavalli. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.